A Large-Scale Fault-Tolerant Distributed Software-Build Process

A large software system can be compiled and linked more quickly if its build process is distributed across a network of multiple computers. However, large networks are more likely to contain a computer that causes a build to fail. If such a computer can be identified, it can be excluded from participation. Otherwise, if the failed command can be detected, it can be retried on a different computer. We describe our experiences designing, implementing, and maintaining a fault-tolerant distributed build process for an industrial software-development environment. We focus on techniques that augment the capabilities of available distributed-build tools. Our build process produces Hewlett-Packard’s laser printer firmware. Our environment includes hundreds of engineers, about one thousand computers, and about two million lines of code. As an example of the speedup provided by distribution, a forced sequential rebuild of all targets requiring about 155 minutes can be accomplished concurrently in about 35 minutes.